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DETAIL ACTION 



1. 



This communication is responsive to Amendment, filed 10/12/06. 



2. 



Claims 1-35 are pending in this application. This action is made Final. 



Claim Rejections - 35 USC §103 



3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of the 
claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various 
claims was commonly owned at the time any inventions covered therein were made absent any 
evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out 
the inventor and invention dates of each claim that was not commonly owned at the time a later 
invention was made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) 
and potential 35 U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 



4. Claims 1-35 are rejected under 35 U.S.C. 103(a) as being unpatentable over Venkayala et 
al. (US Pub. No. 20030212679), in view of Rosen et al. (US Pat. No. 6,513,025). 
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As to claims 1, 28, 35, Venkayala teaches a method of searching data in databases using 
an ensemble of models (i.e. seeds models, [0018]), said method comprising: 

ordering models (i.e. scoring of models, [0017]) within said ensemble in order of 
prediction accuracy (i.e. Trained model 1 10 may also be evaluated and adjusted in order to 
improve the quality, i.e. prediction accuracy, of the model, [0019]), with the most accurate model 
being first in said order (i.e. a topmost category including a class value having a highest 
associated probability, [0010]) ([0008-0009, 0018, 0019, 0035]); 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. how 
much confidence may be placed in the prediction, [0023], the selected class values are those 
meeting the selection criteria presented in prediction parameters, [0024]), wherein said sub- 
ensemble in said order prediction accuracy ([0010, 0035]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). But Venkayala does not expressly 
teach "models are joined together in said sub-ensemble". 

Rosen teaches "models are joined together in said sub-ensemble" (i.e. The most 
dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 
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It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. 

One of ordinary skill in the art would be motivated to. make this combination in order to 
select the most appropriate classification model, and the prediction of that classification model is 
then accepted in view of Rosen, as doing so would give the added benefit of coalescing the 
predictions of the base models by learning the relationships between those predictions and the 
correct prediction, identifying a specific classification model as the one responsible for 
producing a final prediction and to including a simple explanation of why the prediction was 
made, and revising the selected model as the models change over time as taught by Rosen (col. 2, 
lines 31-42). 

As per claim 8, Venkayala teaches a method of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said method comprising: 

ordering models (i.e. scoring of models, [0017]) within said ensemble in order of 
prediction accuracy (i.e. Trained model 110 may also be evaluated and adjusted in order to 
improve the quality, i.e. prediction accuracy, of the model, [0019]), with the most accurate model 
being first in said order (i.e. a topmost category including a class value having a highest 
associated probability, [0010]) ([0008-0009, 0018, 0019, 0035]); 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. how 
much confidence may be placed in the prediction, [0023], the selected class values are those 
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meeting the selection criteria presented in prediction parameters, [0024]), wherein said sub- 
ensemble in said order prediction accuracy, such that said sub-ensemble include only the most 
accurate models (i.e. a topmost category including a class value having a highest associated 
probability, [0010]) ([0008, 0009, 0035]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). Venkayala does not expressly teach 
"models are joined together in said sub-ensemble". 

However, Rosen teaches "models are joined together in said sub-ensemble" (i.e. The 
most dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 

It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. 

One of ordinary skill in the art would be motivated to make this combination in order to 
select the most appropriate classification model, and the prediction of that classification model is 
then accepted in view of Rosen, as doing so would give the added benefit of coalescing the 
predictions of the base models by learning the relationships between those predictions and the 
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correct prediction, identifying a specific classification model as the one responsible for 
producing a final prediction and to including a simple explanation of why the prediction was 
made, and revising the selected model as the models change over time as taught by Rosen (col. 2, 
lines 31-42). 

As per claim 15, Venkayala teaches a method of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said method comprising: 

performing training (i.e. training/model building, [0019]) comprising: 

ordering models (i.e. scoring of models, [0017]) within said ensemble in order of 
prediction accuracy (i.e. Trained model 110 may also be evaluated and adjusted in order to 
improve the quality, i.e. prediction accuracy, of the model, [0019]), with the most accurate model 
being first in said order (i.e. a topmost category including a class value having a highest 
associated probability, [0010]) ([0008-0009, 0018, 0019, 0035]); 

forming sub-ensembles (i.e. model apply, [0017]), wherein said sub-ensemble in said 
order of prediction accuracy ([001 0, 0035]); 

calculating confidence values of each of said sub-ensembles (i.e. to generate one or more 
scores for each row of data in scoring data. The scores for each row of data indicate how closely 
the row of data matches attributes of the model, how much confidence may be placed in the 
prediction, [0023]); and making a prediction comprising: 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. The 
selected class values are those meeting the selection criteria presented in prediction parameters, 
[0024]) ([0010, 0035]); and 
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applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]). 

Although Venkayala teaches the step of forming sub-ensembles, wherein said sub- 
ensembles in said order of prediction accuracy (i.e. model apply, [0017]), Venkayala does not 
expressly teach "joining different number of models together to form sub-ensembles". 

Rosen teaches "joining different number of models together to form sub-ensembles" (i.e. 
The most dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 

It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include joining different numbers of models together to form sub-ensembles, wherein models are 
joined together in said sub-ensemble in said order of prediction accuracy as taught by Rosen. 

One of ordinary skill in the art would be motivated to make this combination in order to 
select the most appropriate classification model, and the prediction of that classification model is 
then accepted in view of Rosen, as doing so would give the added benefit of coalescing the 
predictions of the base models by learning the relationships between those predictions and the 
correct prediction, identifying a specific classification model as the one responsible for 
producing a final prediction and to including a simple explanation of why the prediction was 
made, and revising the selected model as the models change over time as taught by Rosen (col. 2, 
lines 31-42). 
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As per claim 21, Venkayala teaches a service of searching data in databases using an 
ensemble of models (i.e. seeds models, [0018]), said service comprising: 

ordering models (i.e. scoring of models, [0017]) within said ensemble in order of 
prediction accuracy (i.e. Trained model 110 may also be evaluated and adjusted in order to 
improve the quality, i.e. prediction accuracy, of the model, [0019]), with the most accurate model 
being first in said order (i.e. a topmost category including a class value having a highest 
associated probability, [0010]) ([0008-0009, 0018, 0019, 0035]); 

selecting a sub-ensemble of said models that meets a given level of confidence (i.e. how 
much confidence may be placed in the prediction, [0023], the selected class values are those 
meeting the selection criteria presented in prediction parameters, [0024]), wherein said sub- 
ensemble in said order prediction accuracy ([0010, 0035]); and 

applying said sub-ensemble, in place of said ensemble, to an example to make a 
prediction (i.e. The selected class values, which are included in multi-category apply output, 
[0024]) ([0010, 0035]). 

Venkayala teaches the step of forming sub-ensembles, wherein said sub-ensembles in 
said order of prediction accuracy (i.e. model apply, [0017]). But Venkayala does not expressly 
teach "models are joined together in said sub-ensemble". 

However, Rosen teaches "models are joined together in said sub-ensemble" (i.e. The 
most dependable classification model, that is the classification model associated with the 
dependability model indicating the highest dependability for the unlabeled example, is chosen to 
classify each unclassified example (col. 4, line 66 to col. 5, line 9). 
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It would have been obvious to one of ordinary skill of the art having the teaching of 
Venkayala and Rosen at the time the invention was made to modify the system of Venkayala to 
include wherein models are joined together in said sub-ensemble in said order of prediction 
accuracy as taught by Rosen. 

One of ordinary skill in the art would be motivated to make this combination in order to 
select the most appropriate classification model, and the prediction of that classification model is 
then accepted in view of Rosen, as doing so would give the added benefit of coalescing the 
predictions of the base models by learning the relationships between those predictions and the 
correct prediction, identifying a specific classification model as the one responsible for 
producing a final prediction and to including a simple explanation of why the prediction was 
made, and revising the selected model as the models change over time as taught by Rosen (col. 2, 
lines 31-42). 

As to claims 2, 9, 16, 22, 29, Venkayala teaches said sub-ensemble includes fewer 
models than said ensemble (i.e. The selection criterion may comprise one of a topmost category 
including a class value having a highest associated probability, top N categories including N 
class values having highest associated probabilities, bottom N categories including N class 
values having lowest associated probabilities, or a set of select class values specified by the user 
and their associated probabilities and ranks, [0010]). 

As to claims 3, 10, 17, 23, 30, Venkayala teaches said confidence is a measure of how 
closely results from said sub-ensemble will match results from said ensemble (i.e. The scores for 
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each row of data indicate how closely the row of data matches attributes of the model, how much 
confidence may be placed in the prediction, how likely each output, [0049]). 

As to claims 4, 11, 18, 24, 31, Venkayala teaches the size of each sub-ensemble is 
different and has a potentially different level .of confidence (i.e. The selection criterion may 
comprise one of a topmost category including a class value having a highest associated 
probability, top N categories including N class values having highest associated probabilities, 
bottom N categories including N class values having lowest associated probabilities, or a set of 
select class values specified by the user and their associated probabilities and ranks, [0010]). 

As to claims 5, 12, 19, 25, 32, Venkayala teaches the size of said ensemble is fixed (i.e. 
the selection criteria may include a limit on the number of class values that are to be selected, 
[0050]). 

As to claims 6, 13, 20, 26, 33, Venkayala teaches as the level of confidence is raised, a 
sub-ensemble that has more models will be selected in said selecting process (i.e. The selection 
criteria may be defined by desired results data, [0024]), and as the level of confidence is lowered, 
a sub-ensemble that has fewer models will be selected in said selecting process (i.e. The 
selection criterion may comprise one of a topmost category including a class value having a 
highest associated probability, top N categories including N class values having highest 
associated probabilities, bottom N categories including N class values having lowest associated 
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probabilities, or a set of select class values specified by the user and their associated probabilities 
and ranks, [0010]). 



As to claims 7, 14, 27, 34, Venkayala teaches before said selecting, calculating 
confidence values of different sub-ensembles (i.e. to generate one or more scores for each row of 
data in scoring data. The scores for each row of data indicate how closely the row of data 
matches attributes of the model, how much confidence may be placed in the prediction, how 
likely each output, [0049]). 

Response to Arguments 
5. Applicant's arguments filed 10/12/06 have been fully considered but they are not 
persuasive. 

A. Applicant argues that the "class values" of Venkayala have nothing to do with a sub- 
ensemble of models. Moreover, the class values are not selected prior to the application of the 
models, wherein the selected class values are applied in place of the models. 

The Examiner respectfully disagrees for the following reasons: 

a) The Venkayala teaches "ensemble", and "sub-ensemble", "ordering" as follows: 

According to Applicants* Specification, the term "ensemble" and "sub-ensemble" are 
disclosed as "First the invention performs training. This training orders models within the 
ensemble in order of prediction accuracy with the most accurate model being first, and joined 
together in the sub-ensemble in order of prediction accuracy (Instant Specification, Page 1). 
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Analogously, Venkayala' s teaches all such steps as "generating multi-category apply 
output with a plurality of predicted class values and their associated probabilities based on the 
received input data and a selection criterion" ([0007]), and "The selection criterion may comprise 
one of a topmost category including a class value having a highest associated probability, top N 
categories including N class values having highest associated probabilities" ([0008]), wherein: 

(a) multi-category of Venkayala equates to the ensemble; 

(b) a particular category in the multi-category of Venkayala equates to the model; 

(c) a topmost category of Venkayala equates to orders models; 

(d) a top most category of Venkayala equates to the most accurate model being first; 

(e) a topmost category including a class value of Venkayala equates to sub-ensemble. 

It is recognized that a class value itself is not a sub-ensemble as Applicants argue; rather, 
it is "a topmost category including a class value" that equivalent to Applicants' "sub- 
ensemble". 

b) The Venkayala teaches "selection a sub-ensemble" as "The selection criterion may 
comprise one of a topmost category including a class value" (See [0007]). 

Note that a topmost category including a class value of Venkayala equates to sub- 
ensemble. 

c) The Venkayala teaches "applying said sub-ensemble, in place of said ensemble, to 
an example to make a prediction" as " A score/prediction is a category associated with 
probability as the result of applying to a supervised model a record whose target value is 
unknown. A single-target apply operation produces the target value (or category) whose 
probability is the highest among the all target values" ([0030]), wherein: 
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a category associated with probability of Venkayala equates to sub-ensemble. 
a supervised model a record whose target value is unknown of Venkayala equates to an 
example. 

A score/prediction of Venkayala equates to make a prediction. 
As mentioned in (a), a topmost category including a class value (See (e)) is also a 
category associated with probability. 

B. Applicant argues that the "scoring of models" in Venkayala does not teach ordering, 
ranking, or otherwise sorting models. Instead, the scoring step applies a trained model to make 
predictions based on data. 

The examiner respectfully disagree. 

As aforementioned in (c), a topmost category ([0008]) of Venkayala equates to orders 
models; 

C. Applicants argue that the generated output having multiple class values has nothing to 
do with a sub-ensemble of models that is used in place of the ensemble of models. Instead the 
output 122 having multiple class values is produced after the application of the model 110. 

Venkayala teaches " applying said sub-ensemble, in place of said ensemble, to an 
example to make a prediction " as " A score/prediction is a category associated with probability 
as the result of applying to a supervised model a record whose target value is unknown. A 
single-target apply operation produces the target value (or category) whose probability is the 
highest among the all target values" ([0030]) wherein: 

a category associated with probability of Venkayala equates to sub-ensemble. 
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a supervised model a record whose target value is unknown of Venkayala equates to an 
example. 

A score/prediction of Venkayala equates to make a prediction. 

Also a topmost category including a class value (See (e)) corresponds to a category 
associated with probability. 

D. Applicants argue that the independent claims 1, 8, 15, 21, 28, and 35 defines 
"selecting a sub-ensemble' 9 (of the ensemble of models) and "applying said sub-ensemble". 

Similarly, Venkayala does teach selecting a sub-ensemble as "The selection criterion 
may comprise one of a topmost category including a class value" (See [0007]). 

Note that a topmost category including a class value of Venkayala equates to sub- 
ensemble. 

Further, Venkayala teach the step of applying to the model after the step of selecting a 
topmost category including a class value as "A score/prediction is a category associated with 
probability as the result of applying to a supervised model a record whose target value is 
unknown. A single-target apply operation produces the target value (or category) whose 
probability is the highest among the all target values" ([0030]), wherein: 

a category associated with probability of Venkayala equates to sub-ensemble. 

a supervised model a record whose target value is unknown of Venkayala equates to an 
example. 

A score/prediction of Venkayala equates to make a prediction. 

And, a topmost category including a class value (See (e)) is also a category associated 
with probability. 
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Arguments as raised are moot since all claim limitations relevant to this issue have been 
addressed accordingly. 
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Conclusion 



THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Miranda Le whose telephone number is (571) 272-41 12. The 
examiner can normally be reached on Monday through Friday from 8:30 AM to 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John R. Cottingham, can be reached on (571) 272-7079. The fax number to this Art 
Unit is 571-273-8300. 

Any inquiry of a general nature or relating to the status of this application should be 
directed to the Group receptionist whose telephone number is (703) 305-3900. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov . Should you have questions on access to the Private 
PAIR system, contact the Electronic Business Center (EBC) at 866-21 7-91 97 (toll-free). 

^yt/UM>^ 

Miranda Le 
December 28, 2006 




